Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications

ABSTRACT

An integrated display and camera without simple or compound lenses comprising interleaved light-emitting and light-sensing/photosensor elements. Image formation without use of a simple or compound lens can be obtained via software algorithms operating on light-sensing measurements, for example implementing a two-dimensional deconvolution operation defined by the transfer function imposed by a microoptic array. For example, a microoptic aperture array can be configured to comprise localized optical overlaps which are readily deconvolved or otherwise solved for by an algorithm. Associated microoptic structures can be as simple as apertures or can be more complex, for example including microlenses. Display and photosensor elements can include color capabilities. Light-sensing elements can be similar, nearly, or essentially identical in structure and/or composition to the light-emitting element. Light-emitting elements can comprise an Organic Light Emitting Diode, and can be configured for light-detection modalities, for example emitting light and detecting light simultaneously. Applications include providing eye contact in video conferencing.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Divisional of U.S. patent application Ser. No. 09/601,384filed Jul. 27, 2000, now abandoned which was an application filed under35 U.S.C. §371 of International Patent Application Ser. No.PCT/US1999/001789, filed Jan. 27, 1999, which claims the benefit of U.S.Provisional Application Ser. No. 60/072,762, filed Jan. 27, 1998, thedisclosures of all of which are incorporated herein by reference.

1. BACKGROUND OF THE INVENTION

1.1 Field of the Invention

The present invention relates generally to multimedia conferencingsystems, and more particularly to multimedia-enabled communication andcomputing devices. Still more particularly, the present invention is adevice for providing real-time multimedia conferencing capabilities toone or more companion computers or on a stand-alone basis.

1.2 Background

Early computers were large, clumsy, difficult-to-operate and unreliableroom-sized systems shared within a single location. Similarly, earlyvideo and graphics teleconferencing systems suffered from the samedrawbacks, and were also shared within a single location. With regard tocomputers, technological innovations enabled the advent of desktop“personal computers.” Relative to teleconferencing systems, newtechnologies were also introduced, such as those described in U.S. Pat.No. 5,617,539, entitled “Multimedia Collaboration System with SeparateData Network and A/V Network Controlled by Information Transmitting onthe Data Network,” that brought high-quality, reliable video andgraphics teleconferencing capabilities to a user's desktop. In bothearly desktop personal computers and conferencing systems, there wereand remain many incompatible implementations.

Digital technology innovations targeted at working in conjunction withmarket forces gave rise to standardized desktop computer platforms, suchas Microsoft/Intel machines and Apple machines, which have existing andstrengthening unifying ties between them. The standardization ofconverging platforms unified fragmentations that existed within thecomputer hardware and software industries, such that immense economiesof scale lowered the per-desktop development and manufacturing costs.This in turn greatly accelerated desktop computer usage and promoted theinterworking between applications such as work processing, spreadsheet,and presentation tool applications that freely exchange data today. As aresult, businesses employing such interworking applications became moreefficient and productive. The push for greater efficiency has fueled thedevelopment of additional innovations, which further led to developmentssuch as the explosion in electronic commerce as facilitated by theworld-wide Internet.

Relative to present-day desktop conferencing, there are many networkingapproaches characterized by varying audio/video (A/V) quality andscalability. In recent years, customers have assumed a wide range ofpositions in their investments in such technologies. At one end of thisrange, various types of dedicated analog A/V overlay networks exist thatdeliver high-quality A/V signals at a low cost. At another end of thisrange are local area data network technologies such as switched Ethernetand ATM data hubs that function with high-performance desktop computers.These desktop computers and data networking technologies currentlysupport only lower-quality A/V capabilities at a relatively high cost.Despite this drawback, these desktop computers and data networkingtechnologies are believed to be the preferred path for eventuallyproviding high-quality A/V capabilities at a low cost. Other A/Vnetworking solutions, such as ISDN to the desktop, also lie in thisrange.

Within each of many separate networked A/V technology “islands,” variousapproaches toward providing multimedia applications such asteleconferencing, video mail, video broadcast, video conferencerecording, video-on-demand, video attachments to documents and/or webpages, and other applications can be performed only in fragmented wayswith limited interworking capability. For many years, it has beenprojected that the desktop computer industry and the data networkingindustry will solve such fragmentation and interworking problems, andeventually create a unified, low-cost solution. Several generations ofthese technologies and products have consistently fallen short ofsatisfying this long-felt need. Furthermore, it is likely to bedisadvantageous to continue to rely upon the aforementioned industriesto satisfy such needs. For example, if the introduction of today'sstandardized multi-method fax technology had been held back by those whomaintain that the idea that all electronic text should only be computerASCII (as advocated, for example, by M.I.T. Media Lab DirectorNegroponte), a great amount of the fax-leveraged domestic andinternational commerce that has occurred since the early 1980's may nothave occurred. Desktop multimedia technologies and products arecurrently in an analogous position, as it is commonly accepted that itwill be only the desktop computer and data networking industries that atsome point in the future will make high-quality networked A/V widely anduniformly available, and at the same time it is doubtful that this willoccur any time soon.

What is sorely needed, given the pace and market strategies of thedesktop computer and data networking industries, is an integration ofseparate technology and application islands into a single low-cost,manufacturable, reliable real-time multimedia collaboration apparatuscapable of supporting a wide range of A/V networking technologies; A/Vapplications; and A/V and data networking configurations in a widevariety of practical environments. A need also exists for a design orarchitecture that makes such an apparatus readily adaptable to futuretechnological evolution, such that the apparatus may accommodateevolving or new families of interrelated standards.

2. SUMMARY OF THE INVENTION

This invention relates to a multimedia device for use in multimediacollaboration apparatus and systems. Such apparatus and systems alsotypically contain processing units, audio reception and transmissioncapabilities, as well as video reception and transmission capabilities.The reception and transmission capabilities allow analog audio/videosignal transfer over UTP wires for audio transmit/receive. Furtherincluded in these capabilities is audio/video signal transfer viaencoding both audio and video signals on a single set of UTP wires, forexample, through frequency modulation (FM). The video receptioncapabilities may include support for a primary digital video stream andan auxiliary digital video stream. The reception, transmission,encoding, and decoding capabilities could exist in a single packaging.This or another single packaging can support a plurality of multimedianetwork signal formats, including analog plus digital or all digital.Different wire pair combinations could also be supported, such as 10 and100 Megabit-per-second (MBPS) Ethernet, as well as Gigabit Ethernet, viaUnshielded Twisted Pair (UTP) wiring. Other embodiments could includesupport for other or additional networking protocols, such asAsynchronous Transfer Mode (ATM) networking. AV reception capabilitiesinclude adaptive stereo echo-canceling capabilities and syntheticaperture microphone capabilities.

In addition, this invention may include a single packaging allowing forstereo echo-canceling capabilities. The invention also includessynthetic aperture microphone capabilities, such as capabilities forprogrammably adjusting a position of a spatial region corresponding tomaximum microphone audio sensitivity. The synthetic aperture microphonecapabilities typically are implemented through an audio signalprocessing unit and a plurality of microphones.

This system further embodies multiport networking capabilities in whicha first port couples to a multimedia network which can carry multimediasignals in multiple format, and a second port couples to a set ofcomputers. These multiport networking capabilities also include datapacket destination routing.

Moreover, the invention includes a memory in which an operating systemand application software having internet browsing capabilities resides.A graphical user interface is included in the invention with I/Ocapabilities that support graphical manipulation of a cursor andpointing icon.

The multimedia apparatus also includes a display device havingintegrated image capture capabilities. Typically, the display device isa single substrate upon which display elements and photosensor elementsreside. The display device has display elements interleaved with aplurality of photosensor elements in a planar arrangement. Further, thedisplay elements may be integrated with the photosensor elements. Thedisplay elements are typically optically semitransparent.

Photosensor elements typically occupy a smaller area than the displayelements and are fabricated with different geometries such that thenonluminent spacing between display elements is reduced. Also, thephotosensor elements and sets of display elements are fabricated withoptical structures to minimize perceived areas of nonluminescencebetween a set of displayed pixels.

Among other characteristics of the photosensor elements are: (1) aplurality of photosensor elements in the display device areindividually-apertured, and (2) a set of photosensor elements in thedisplay device includes dedicated microoptic structures. Also, imageprocessing capabilities are coupled to the photosensor elements in thedisplay device.

The display device can operate to display an image on a screen whilecapturing external image signals. This is done by outputting displaysignals to a set of display elements while capturing external imagesignals using a set of photosensor elements. These sets of display andphotosensor elements occupy different lateral regions across the planeof the display device. The first set of display elements comprises atleast one display line across the screen, and the first set ofphotosensor elements comprises a photosensor line across the screen.Display lines and photosensor lines may be scanned in a temporally orspatially separate manner.

The device performs a set of optical image processing operations byreceiving external image signals through a set of apertures or a set ofmicrooptic elements. The device then outputs an electrical signal ateach photosensor element within a set of photosensor elementscorresponding to the set of apertures. These electrical signals havemagnitudes dependent upon the light intensity detected by thephotosensor elements.

3. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram of a multimedia collaborationdevice constructed in accordance with the present invention.

FIG. 2 is a high-level perspective view illustrating a box package forthe multimedia collaboration device.

FIG. 3 is a high-level drawing of a plug-in card package for themultimedia collaboration device, which also includes a bus interface.

FIG. 4 is a perspective view of a stand-alone package for the multimediacollaboration device, which includes a camera, a display, a microphonearray, and speakers.

FIG. 5 is a block diagram of a first embodiment of a multimediacollaboration device constructed in accordance with the presentinvention, and which provides primary and auxiliary (AUX) support foranalog audio/video (A/V) input/output (I/O), and further providessupport for networked digital streaming.

FIG. 6 is a block diagram of a second embodiment of a multimediacollaboration device, which provides primary support for analog audioI/O and digital visual I/O, and further supports analog and digitalauxiliary A/V I/O, plus networked digital streaming.

FIG. 7 is a block diagram of a third embodiment of a multimediacollaboration device, which provides primary support for analog audioI/O and digital visual I/O, support for digital auxiliary A/V I/O, andsupport for networked digital streaming.

FIG. 8 is a block diagram of an adaptive echo-canceled stereo microphoneand stereo speaker arrangement within an audio signal conditioning unitof the present invention.

FIG. 9 is a block diagram of an adaptive echo-canceled mono-outputsynthetic aperture microphone arrangement, assuming stereo speakers,within the audio signal conditioning unit, which is of particular valuein noisy environments such as office cubicles or service depot areas.

FIG. 10 is an illustration showing an exemplary localized primaryhot-spot, within which the synthetic aperture microphone has enhancedsensitivity to sound waves produced by a user.

FIG. 11 is an illustration showing exemplary primary hot-spotdirectivity, where the synthetic aperture microphone captures or rejectsdirectionally-specific sound energy from a user within a primaryhot-spot that is offset relative to that shown in FIG. 10.

FIG. 12 is an illustration showing exemplary reflected speech energyrejection by the synthetic aperture microphone.

FIG. 13 is an illustration showing exemplary ambient audio noiserejection by the synthetic aperture microphone.

FIG. 14 is a block diagram of a first embodiment of a first and a secondmultimedia network interface provided by the present invention.

FIG. 15 is a block diagram of a second embodiment of a first and asecond multimedia network interface provided by the present invention.

FIG. 16 is an illustration of a first photosensor and display elementplanar interleaving technique.

FIG. 17 is an illustration of an exemplary photosensor element color anddisplay element color distribution scheme.

FIG. 18 is an illustration of a second alternating photosensor anddisplay element interleaving technique, in which photosensor and displayelement geometries and size differentials aid in minimizing pixel pitchand maximizing displayed image resolution.

FIG. 19 is a cross-sectional view showing a full-color pixel arrayintegrated with a photosensor element array upon a common substrate.

FIG. 20 is a cross-sectional view showing an integrated full-colorpixel/photosensor element, which may form the basis of an integrateddisplay element/photosensor element array.

FIG. 21 is a cross-sectional view of a first full-coloremitter/detector.

FIG. 22 is a cross-sectional view of a second full-coloremitter/detector.

FIG. 23 is a cross-sectional view of a third full-coloremitter/detector.

FIG. 24 is a top-view of an exemplary microoptic layer having differentoptical regions defined therein.

FIG. 25 is an illustration showing individually-apertured photosensorelements capturing light from portions of an object and outputtingsignals to an imaging unit.

4. DETAILED DESCRIPTION

4.1 General Provisions

The present invention comprises a device that provides analogaudio/video and/or digital audio/visual (both referred to herein as A/V)multimedia collaboration capabilities to a user coupled to a multimedianetwork, such as a Multimedia Local Area Network (MLA/V) as described inU.S. Pat. No. 5,617,539 the disclosure of which is incorporated hereinby reference.

The present invention may operate either in conjunction with one or moreuser's computers or in a stand-alone manner, and may support two-wayvideoconferencing, two-way message publishing, one-way broadcasttransmission or reception, one-way media-on-demand applications, as wellas other audio, video, and/or multimedia functionality or operations.The present invention may support such multimedia functionality across awide range of multimedia network implementations, including mixed analogand digital and/or all-digital multimedia networks. When used inconjunction with a companion computer (i.e., desktop, laptop,special-purpose workstation or other type of computer), the presentinvention may operate as a high-performance multimedia processing devicethat offloads potentially computation-intensive multimedia processingtasks from the companion computer.

The present invention unifies several previously segregated or disparateaudio-, video-, and/or multimedia-related technologies in a singlephysical device that supports multiple multimedia applications andmultiple network signal formats and standards. Such technologies mayinclude hardware and/or software that provide audio signal processing,analog-to-digital (A-D) and digital-to-analog (D-A) conversion,compression and decompression, signal routing, signal level control,video conferencing, stored video-on-demand, internet browsing, messagepublishing, and data networking capabilities. Heretofore, thesetechnologies were typically implemented via separate devices and/orsystems that may have operated in accordance with different data orsignal formats and/or standards, and that offered limited ability (ifany) to interface or operate together.

In particular, the multimedia collaboration device described hereinsupports functionality that may include the following:

-   1. Audio signal handling:    -   a) stereo speakers—to provide realistic audio reproduction        capabilities needed for multimedia presentations, music, and        multiport teleconferencing, including support for        three-dimensional sound and audio positioning metaphors;    -   b) adaptive echo-canceled stereo speakers for the environment        and mono or stereo microphone—to provide high-quality, realistic        audio interactions and eliminate echo and/or feedback in        conferencing situations; and    -   c) adaptive echo-canceled mono synthetic aperture microphone—to        significantly improve audio capture performance in noise-prone        or poorly-controlled audio environments, such as office cubicles        or public kiosks.-   2. One or more data networking protocols, where such protocols may    span a range of technological generations. In one embodiment, the    present invention includes built-in support for 10 and 100    Megabit-per-second (MBPS) Ethernet, as well as Gigabit Ethernet, via    Unshielded Twisted Pair (UTP) wiring. Other embodiments could    include support for other or additional networking protocols, such    as Asynchronous Transfer Mode (ATM) networking and Integrated    Services Digital Network (ISDN).-   3. One or more analog A/V signal transmission/reception formats,    where such formats may span various means of:    -   a) Analog A/V signal transfer via a separate pair of wires for        each of audio transmit, audio receive, video transmit, and video        receive (i.e., a total of four sets of UTP wires);    -   b) Analog A/V signal transfer via a single set of UTP wires for        audio/video transmit, plus a single set of UTP wires for        audio/video receive (i.e., a total of two twisted-pairs carrying        analog A/V signals), through frequency modulation (FM) or other        multiplexing techniques;    -   c) Analog A/V signal transfer via encoding both audio and video        signals on a single set of UTP wires, for example, through FM or        other multiplexing methods and perhaps 2-wire/4-wire electronic        hybrids; and    -   d) Any of the above approaches that carry the analog A/V signals        on the same wire pairs as used by data networking circuits        (through the use of FM or other modulation techniques).

Either of the above analog A/V signal transfer formats allow the use ofa single conventional data network connector for carrying both analogA/V and data networking signals. For example, a standard 8-wire RJ-45connector can support 10 and/or 100 MBPS Ethernet in conjunction withanalog A/V signal transfer, using two twisted pairs for Ethernetnetworking and two twisted pairs for A/V signal transfer. In the eventthat data networking is implemented via a protocol for which asufficient number of connector pins or wires are unavailable for A/Vsignal transfer, such as Gigabit Ethernet, which conventionally utilizesthe entire physical capacity of an RJ-45 connector, the presentinvention may include an additional connector or coupling for analog A/Vsignal transfer.

-   4. Digital multimedia streaming I/O, transmitted to and/or received    from a multimedia network and/or a companion computer, as further    described below.-   5. Internal A/V signal encoding and decoding capabilities to support    A/V compression formats such as MPEG 1/2/4, JPEG, H.310, H.320,    H.323, QuickTime, etc.-   6. Internal data routing capabilities, through which data packets,    cells, or streams may be selectively transferred among a multimedia    network, the present invention, and/or a set of companion computers.-   7. Multimedia call and connection control protocols, such as    described in U.S. Pat. No. 5,617,539.-   8. Internet browsing and multimedia internet message transfer    capabilities.-   9. Data sharing and/or application sharing protocols.-   10. Network configuration and/or network traffic monitoring    capabilities.

Through the combination of the data routing, internal encoding/decoding,and/or digital streaming capabilities, the present invention may operateas a multimedia processing device that offloads potentiallycomputationally-intensive multimedia processing tasks from a companioncomputer. Use of the present invention to reduce a companion computer'sprocessing burden can be particularly advantageous in real-timemultimedia situations. The present invention may further provide anolder or outdated computer with comprehensive real-time multimediacollaboration capabilities, as described below. Additionally, thepresent invention may operate as a stand-alone device, such as aself-contained internet or intranet appliance having real-timemultimedia capabilities, and/or an ISDN video teleconferencing terminal.

The present invention also may advantageously incorporate newtechnologies, including an integrated camera/display device as describedin detail below. Furthermore, the present invention provides support fortechnology and standards evolution by 1) facilitating the use ofstandard plug-in and/or replaceable components, which may be upgraded orreplaced over time; 2) providing designed-in support forrecently-developed technologies that are likely to gain widespread use,such as switched 10 MBPS full-duplex internet, 100 MBPS switchedEthernet, ATM, or Gigabit Ethernet (as well as interim-value networkssuch as ISDN); and 3) providing for upgradability via software and/orfirmware downloads. The present invention may additionally implementparticular capabilities via reconfigurable or reprogrammable logicdevices, such as Field Programmable Gate Arrays (FPGAs). Updatedconfiguration bitstreams can be downloaded into these reconfigurabledevices to provide hardware having upgraded or new capabilities.

4.2 High-Level Architecture and Packaging Options

FIG. 1 is a high-level block diagram of a multimedia collaborationdevice 100 constructed in accordance with the present invention. Themultimedia collaboration device 100 comprises a preamplifier and bufferunit 102; an audio signal conditioning unit 104; a switching unit 106;an Unshielded Twisted Pair (UTP) transceiver 108; a pair splitter 110; arouting unit 112; an encoding/decoding unit 116; a processor set 118; amemory 120; an input device interface 130; a companion computer port136; and a building or premises network port 138.

The premises network port 138 facilitates coupling to premises- orbuilding-based UTP wiring that forms a portion of a multimedia network60. In one embodiment, the premises network port 138 comprises aconventional network coupling, such as an RJ-45 connector. The companioncomputer port 136 facilitates coupling to one or more host or companioncomputers 50, such that the present invention can offload real-timemultimedia processing tasks from a companion computer 50 and/or providea pass-through for data packet exchange between a host computer 50 andthe multimedia network 60. In one embodiment, the companion computerport 136 comprises a conventional network coupling that is compatiblewith the premises network port 138. In another embodiment, the premisesnetwork port 138 may employ a more sophisticated or modern protocol thanthat used by the companion computer port 136. In yet another embodiment,a host or companion computer may access the multimedia collaborationdevice 100 via the premises network port 138, and hence such anembodiment may not include a separate companion computer port 136. It isalso possible for the present invention to communicate with a host orcompanion computer 50 over the data networking ports 136, 138 for use inrunning Graphical User Interfaces (GUIs) or coordinating withapplication processes executing on the host or companion computer 50.

The preamplifier and buffer unit 102 receives A/V signals from a leftand a right microphone 140.1, 140.2 and a camera 142, and transmits A/Vsignals to a left and a right speaker 144.1, 144.2 and a display device146. The preamplifier and buffer unit 102 can additionally send andreceive A/V signals via a set of auxiliary (AUX) A/V ports 148, whichcould couple to a device such as a Video Cassette Recorder (VCR).

As elaborated upon below, the audio signal conditioning unit 104provides volume control functionality in conjunction with echo-canceledstereo microphone or mono synthetic aperture microphone capabilities. Inone embodiment, the echo-canceled stereo microphone and mono syntheticaperture microphone capabilities may be implemented in a singlemode-controlled Digital Signal Processor (DSP) chip, in a manner thatmay facilitate user-selectivity between these two types of microphonefunctionality. If the microphone array 140.1, 140.2 includes more thantwo microphones, it may be desirable to employ DSP techniques tosynthesize a stereo synthetic aperture microphone. Further multiplemicrophone processing modes, such as stochastic noise suppression forextreme noise environments, can also be included.

In the present invention, transfer of incoming and/or outgoing A/Vsignals between a variety of sources and/or destinations is required,including the microphones 140.1, 140.2, the camera 142, the speakers144.1, 144.2, the display device 146, other A/V or I/O devices, thepremises network port 138, and/or the companion computer port 136.Signal transfer pathways for such sources and destinations mayultimately be analog or digital in nature. To meet these switchingneeds, the multimedia collaboration device employs the switching unit106, which selectively routes analog A/V signals associated with themicrophones 140.1, 140.2, the camera 142, the speakers 144.1, 144.2, thedisplay device 146, and/or other devices to or from the analog A/V UTPtransceiver 108 and/or the encoding/decoding unit 116. Theencoding/decoding unit 116 may also perform any required conversionbetween analog and digital formats.

As further described below, the analog A/V UTP transceiver 108 providesan analog signal interface to the pair splitter 110, which separatesdata networking and analog A/V signals. In many cases this signalseparation is most easily accomplished by selectively separating wiresor wire pairs, but may also include the use of passive (or equivalent)wire switching arrangements and programmable Frequency DivisionMultiplexing (FDM) modulators and demodulators. As indicated earlier,the encoding/decoding unit 116 performs conversions between analog anddigital signal formats, and as such also compresses and decompresses A/Vsignals. Although not shown, those skilled in the art will understandthat an ISDN transceiver, inverse multiplexer, network connector, Q.931call control, etc. . . . can be introduced into the architecture to addsupport for ISDN. The processor set 118 controls the operation of themultimedia collaboration device 100, and performs data networkcommunication operations. In conjunction with operating system and othersoftware resident within the memory 120, the processor set 118 mayprovide graphic overlay capabilities on a video image so as to implementany GUI capabilities. These GUIs may facilitate control over theoperations of the present invention, and may further provide internetbrowsing capabilities, as described in detail below. The routing unit112 performs network packet exchange operations between the premisesnetwork port 138, the companion computer port 136, and the processingunit 118, where such packets may include data, portions of, or entiredigital AV streams, and/or network configuration or traffic monitoringinformation. Finally, the input device interface 130 may provideauxiliary mouse and keyboard ports 132, 134, and may also support aninternal local geometric pointing input device as described below.

Particular groupings of the aforementioned elements may be packaged invarious manners so as to match particular deployment settings. Forexample, selected element groupings may reside within or upon aperipheral box package, computer-bus-compatible card, or housing 150,where such element groupings may include various A/V transducers. Thenature of the selected package 150, and the manner in which theaforementioned elements are incorporated therein or thereupon asintegrated, modular, plug-in, and/or other types of components, isdependent upon the manner in which the present invention is employed,and may be subject to or adaptive to evolving market forces and embeddedlegacy equipment investments. Three exemplary types of packages aredescribed in detail hereafter.

FIG. 2 is a high-level perspective view illustrating a box package 160for the multimedia collaboration device 100. This illustrative boxpackage 160 comprises a housing 162 having a control panel 164 and acable panel 182. The control panel 164 includes an audio mode control166; a microphone/speaker/headset selector 168; a microphone mutecontrol 170; a hold/resume control 172; AUX video and audio inputs 174,176; a telephone add/remove control 178; and a speaker/earphone volumecontrol 180. The audio mode control 166 facilitates user-selectionbetween stereo microphone and synthetic aperture microphone operation,as further described below. The microphone/speaker/headset selector 168provides for user-selection of different audio input/output interfaces,and the microphone mute control 170 facilitates user control over audioinput muting. The hold/resume control 172 pauses or resumes audio inputsin response to user-selection. The AUX video and audio inputs 174, 176respectively facilitate video and audio input from various sources. Thetelephone add/remove control 178 provides control of the insertion of anoptional bridge or coupling to a telephone line for two-way audiocontact with an addition of third-party telephone user. The supportingelectrical couplings would provide for standard telephone loop-through.In one embodiment, the telephone add/remove control 178 includesconventional telephone line echo cancellation circuitry to remove theundesired transmit/receive coupling effects introduced by telephoneloops. Finally, the speaker/earphone volume control 180 controls theamplitude of an audio signal delivered to speakers or an earphone (inaccordance with the setting of the microphone/speaker/headset selector168). Some implementations may include separate volume controls forspeakers, earphones, and/or auxiliary audio I/O.

The cable panel 182 on the box package 160 includes inputs and outputsthat facilitate coupling to a camera/microphone cable 184; a premisesUTP cable 186; left and right speaker cables 188, 190; a video monitoror video overlay card cable 192; and a UTP computer networking cable194.

The box package 160 is suitable for use with a companion desktop orportable computer, and could reside, for example, underneath, atop, oradjacent to a computer or video monitor. Furthermore, a single boxpackage 160 may be used to provide a plurality of companion computers 50with multimedia collaboration capabilities, for example, in a smalloffice environment.

Those skilled in the art will understand that the above combination offeatures is illustrative and can be readily altered. Those skilled inthe art will also understand that in an alternate embodiment, the boxpackage 160 could include a built-in microphone or microphone array, aswell as one or more speakers. Furthermore, those skilled in the art willunderstand that one or more controls described above could beimplemented via software.

FIG. 3 is a suggestive high-level drawing showing the format of aplug-in card package 200 for the multimedia collaboration device 100.The plug-in card package 200 comprises a circuit board or card 202having a standard interface 204 that facilitates insertion into anavailable slot within a computer. For example, the standard interface204 could comprise plated connectors that form a male PeripheralComponent Interface (PCI) connector, for insertion into a female PCIslot coupled to a PCI bus. The elements comprising the multimediacollaboration device 100 may be disposed upon the card 202 in the formof discrete circuitry, chips, chipsets, and/or multichip modules. Thecard 202 includes inputs and outputs for coupling to a camera/microphonecable 214; left and right speaker cables 206, 208; a premises UTP cable210; and a UTP-to-computer cable 212 that facilitates pass-through ofdata networking signals to an existing data networking card. It isunderstood that conventional PCI bus interface electronics and firmwaremay be added to this configuration. Alternatively, the PCI bus maysimply be used to provide power and electrical reference grounding.

The multimedia collaboration device 100 may include more extensive datanetworking capabilities, capable in fact of supporting essentially allthe networking needs of one or more companion or host computers, asdescribed in detail below. In this variation, the plug-in card package200 may therefore be used to provide a computer into which it isinserted with complete data networking capabilities in addition tomultimedia collaboration capabilities via transfer of data networkingpackets between the interface 204 and the computer, in which case theUTP-to-computer cable 212 may not be necessary. The presence of theplug-in-card package 200 may therefore obviate the need for a separatenetwork interface card (NIC) in market situations in which sufficientevolution stability in data networking technologies exists.

The plug-in card package 200 may be used to provide older orless-capable computers with comprehensive, up-to-date real-timemultimedia collaboration capabilities. Alternatively, the plug-in cardpackage 200 can provide video overlay multimedia capabilities tocomputer systems having a monitor for which a video overlay card isunavailable or difficult to obtain. In the event that video overlaymultimedia capabilities are to be delivered to a display or videomonitor other than that utilized by the companion computer 50, theplug-in card package 200 may include a port that facilitates coupling ofa video monitor or video overlay card cable 192 in a manner analogous tothat shown in FIG. 2. A host computer 50 that incorporates a pluralityof plug-in card packages 200 could be used as a multimedia collaborationserver for other computers, in a manner understood by those skilled inthe art.

Those skilled in the art will additionally understand that one or moreof the physical panel controls described above with reference to the boxpackage 160 would be implemented via software control for the plug-incard package 200.

FIG. 4 is a perspective view of a stand-alone package 300 for themultimedia collaboration device 100 that includes a range ofadvantageous internal A/V transducer configurations. In one deployment,the stand-alone package may be attached, mounted, or placed proximate tothe side of a computer monitor or laptop/palmtop computer, and hence isreferred to herein as a “side-kick” package 300.

The side-kick package 300 provides users with a self-containedhighly-localized multimedia communication interface. The incorporationof the microphone array 304 into the side-kick package 300 assists incontrolling the present invention's superior audio performance relativeto adaptive echo-canceled stereo microphone and adaptive echo-canceledmono synthetic aperture microphone capabilities described below. Theplacement of the camera 306 in close proximity to the flat displaydevice 312 aids in maintaining good user eye contact with a displayedimage, which in turn better simulates natural person-to-personinteractions during videoconferencing. The eye contact can be furtherimproved, and manufacturing further simplified, by an integratedcamera/display device as described below with reference to FIGS. 16through 25.

The side-kick package 300 can be used in conjunction with a companioncomputer 50, or in a stand-alone manner. When used with a companioncomputer 50, the side-kick package 300 eliminates the need to consumecompanion computer screen space with a video window. As a stand-alonedevice, the side-kick package 300 can be used, for example, in officereception areas; public kiosks; outside doorways; or alongsidespecial-purpose equipment for which explicatory, possibly interactiveassistance may be useful, such as a photocopier.

Relative to FIG. 2, like reference numbers designate like elements. Theside-kick package 300 comprises a housing 302 in which the multimediacollaboration device 100 described above plus additional elements suchas an internal shock-mounted microphone array 304; a camera 306 that mayinclude auto-focus, auto-iris, and/or electronic-zoom features;acoustically-isolated stereo speakers 308; a thumbstick mouse or similartype of geometric input device 310; and a flat display device 312 mayreside. The side-kick package 300 may further include display brightnessand contrast controls 314, 316, and/or one or more auxiliary audio levelcontrols 180. Additionally, the side-kick package 300 may include acontrol panel having physical panel controls such as an audio modecontrol 166; a microphone/speaker/headphone selector 168; a microphonemute control 170; a hold/resume control 172; AUX video and audio inputs174, 176; and a telephone add/remove control 178, which function in themanner previously described. Those skilled in the art will understandthat the functions of one or more of the physical controls shown in FIG.4 could be implemented so as to be controlled remotely via software. Insome arrangements, there might not be any physical controls, in whichcase control is facilitated by GUIs executing on one or more companioncomputers 50. Ideally, this embodiment may include both physical andremote software controls so that it can operate as a fully stand-alonedevice as well as a slave device supporting applications running on thecompanion computer 50.

The side-kick package 300 has ports for coupling to a premises UTP cable336 and an optional UTP-to-computer cable 338. The side-kick package 300may also include another connector set 334, which, for example,facilitates coupling to a headset, an auxiliary mouse, and/or anauxiliary keyboard. FIG. 4 additionally depicts an overlay window 340upon the flat display device 312, which may be realized via graphicsoverlay capabilities. The graphics overlay capabilities can implementmenus or windows 340 that can provide a user with information such astext or graphics and which may be selectable via the input device 310,creating internal stand-alone GUI capabilities.

Relative to each package 160, 200, 300 described herein, use of themultimedia collaboration device 100 with one or more companion computers50 to effect digital networked A/V communication advantageously spareseach companion computer 50 the immense computational and networkingburdens associated with transceiving and encoding/decoding A/V streamsassociated with A/V capture and presentation. The invention may alsoincorporate additional video graphics features in any of the packages160, 200, 300 described above, such as telepointing over live videoand/or video frame grab for transference to or from a companion or hostcomputer 50.

While FIG. 1 provides a broad overview of the architecture of thepresent invention, specific architectural details and variousembodiments are elaborated upon hereafter, particularly with referenceto FIGS. 5, 6, and 7.

4.3 Architectural Details

FIG. 5 is a block diagram of a first embodiment of a multimediacollaboration device 10 constructed in accordance with the presentinvention, and which provides primary and auxiliary (AUX) support foranalog A/V, and further provides support for networked digitalstreaming. With reference to FIG. 1, like reference numbers designatelike elements. The embodiment shown in FIG. 5 supports analog A/V, andcomprises the preamplifier and buffer unit 102; the audio signalconditioning unit 104; the A/V switch 106; the analog A/V UTPtransceiver 108; the pair splitter 110; a first and a second digitaltransceiver 111, 135; the routing unit 112; a network interface unit114; an analog-to-digital (A/D) and digital-to-analog (D/A) converter116 a; an A/V compression/decompression (codec) unit 116 b; at leastone, and possibly multiple, processors 118.1, 118.n; the memory 120; theI/O interface 130; and the companion and premises network ports 136,138. An internal bus 115 couples the network interface unit 114, the A/Vcodec 116 b, each processor 118.1, 118.n, the memory 120, and the I/Ointerface 130. Each of the audio signal conditioning unit 104, the A/Vswitch 106, the analog A/V UTP transceiver 108, the routing unit 112,and the A/D-D/A converter 116 a may also be coupled to the internal bus115, such that they may receive control signals from the processors118.1, 118.n.

The preamplifier and buffer unit 102 is coupled to receive left andright microphone signals from a left and right microphone 140.1, 140.2,respectively; and a camera signal from the camera 142. It is understoodthat additional microphones 140.3 . . . 140.x and processing 118 and/orswitching capabilities 106 may be included to enhance the syntheticaperture microphone capabilities described below. The preamplifier andbuffer unit 102 may further receive AUX A/V input signals from one ormore auxiliary A/V input devices such as an external VCR, camcorder, orother device. The preamplifier and buffer unit 102 respectively outputsleft and right speaker signals to a left and a right speaker 144.1,144.2; and a display signal to the display device 146. The preamplifierand buffer unit 102 may also deliver AUX A/V output signals to one ormore auxiliary devices.

The audio signal conditioning unit 104 facilitates the adjustment ofoutgoing audio signal volume in conjunction with providing adaptive echocancelled stereo microphone or mono synthetic aperture microphoneprocessing operations upon audio signals received from the preamplifierand buffer unit 102. FIG. 8 is a block diagram of an adaptiveecho-canceled stereo microphone unit 103 within the audio signalconditioning unit 104. The adaptive echo-canceled stereo microphone unit103 comprises a stereo echo canceller 310 and a stereo volume controlunit 350.

The stereo echo canceller 310 comprises conventional monoaural echocanceller subsystems that function in a straightforward manner readilyapparent to those skilled in the art. This arrangement includes a leftmicrophone/left speaker (LM/LS) adaptive acoustic echo filter model 312;a left microphone/right speaker (LM/RS) adaptive acoustic echo filtermodel 314; a right microphone/left speaker (RM/LS) adaptive acousticecho filter model 316; and a right microphone/right speaker (RM/RS)adaptive acoustic echo filter model 318. It will be readily understoodby those skilled in the art that linear superposition results in stereoecho canceling capabilities for stereo microphones and stereo speakers.

The stereo volume control unit 350 is coupled to a volume adjustmentcontrol such as described above with reference to the various packageembodiments 160, 200, 300 shown in FIGS. 2, 3, and 4, and is furthercoupled to receive the left and right speaker signals. The stereo volumecontrol unit 350 is also coupled to each model 312, 314, 316, 318 inorder to maximize the utilization of DSP arithmetic and dynamic rangethroughout the full range of speaker volume settings. It is understoodthat stereo balance controls can be implemented using the same stereovolume control elements operating in complimentary increments.

The LM/LS and LM/RS models 312, 314 are coupled to receive the left andright speaker signals, respectively. Similarly, the RM/LS and RM/RSmodels 316, 318 are respectively coupled to receive the left and rightspeaker signals 300. Each of the LM/LS, LM/RS, RM/LS, and RM/RS models312, 314, 316, 318 incorporates an adaptive coefficient tapped delayline weighting element coupled to its corresponding microphone 140.1,140.2 and speaker 144.1, 144.2 in a conventional manner. Additionally,the LM/LS and LM/RS models 312, 314 maintain conventional couplings tothe left microphone 140.1 to facilitate initial acoustic environment andsubsequent adaptive acoustic training operations. Similarly, the RM/LSand RM/RS models 316, 318 maintain couplings to the right microphone140.2 to facilitate these types of training operations.

The stereo echo canceller 310 additionally includes a first signalsummer 320 coupled to outputs of the left microphone 140.1, the LM/LSmodel 312, and the LM/RS model 314; plus a second signal summer 322coupled to outputs of the right microphone 140.2, the RM/LS model 316,and the RM/RS model 318. The first signal summer 320 delivers a leftecho-canceled signal to the A/V switch 106, and the second signal summer322 delivers a right echo-canceled signal to the A/V switch 106, in amanner readily understood by those skilled in the art.

In one embodiment, the stereo echo canceller 310 and stereo volumecontrol unit 350 are implemented together via DSP hardware and software.Furthermore, a single DSP may be used to implement the stereo echocanceller 310, the stereo volume control unit 350, and the adaptiveecho-canceled mono synthetic aperture microphone unit 105, which isdescribed below. In an exemplary embodiment, such a DSP may comprise aTexas Instruments TMS320C54x generation processor (Texas InstrumentsIncorporated, Dallas, Tex.).

In the event that a user employs an earphone, headphone set, or AUXaudio device in conjunction with the present invention, as describedabove with reference to the box, card, and side-kick packages 160, 200,300, the stereo echo canceller 310 is placed in a bypassed, inactive, orquiescent state and the DSP and stereo volume control unit 350facilitate normalization and/or volume adjustment in a conventionalmanner as understood by those skilled in the art. Alternatively,separate volume control and/or normalization circuitry could be providedwhen stereo microphones or the stereo echo canceller 310 is not needed.These may be implemented in various ways with respect to the pathsconnecting to the A/V switch.

FIG. 9 is a block diagram of an adaptive echo-canceled mono syntheticaperture microphone unit 105 within the audio signal conditioning unit104. With reference to FIG. 8, like reference numbers designate likeelements. The adaptive echo-canceled mono synthetic aperture microphoneunit 105 comprises the volume control unit 350 plus a synthetic aperturemicrophone processing unit 330, which may include hardware and/orsoftware. The synthetic aperture microphone processing unit 330comprises a synthetic aperture microphone unit 340 which may includehardware and/or software to implement synthetic aperture microphoneprocessing algorithms; a synthetic microphone/left speaker (SM/LS) model332; a synthetic microphone/right speaker (SM/RS) model 334; and asignal summing circuit 336, each coupled in the manner shown.

The synthetic aperture microphone unit 330 is coupled to receive theleft and right microphone signals, and additionally includesconventional adaptive coefficient weighting and training couplings.Taken together, the synthetic aperture microphone unit 330, the leftmicrophone 140.1, and the right microphone 140.2 (plus one or moreadditional microphones that may be present) form a mono-output syntheticaperture microphone. The synthetic aperture microphone unit 330 performsdelay and/or frequency dispersion operations upon the left and rightmicrophone signals to internally create or define an audio receptionsensitivity distribution pattern in a manner readily understood by thoseskilled in the art. The audio reception sensitivity distribution patternincludes one or more spatial regions referred to as “hot-spots,” as wellas a set of spatial regions referred to as “rejection regions.”Typically, a set of one or more “hot-spots” includes a primary hot-spotof maximal audio reception sensitivity that has a particular position ororientation relative to the geometry of the microphone array 140.1,140.2. The rejection regions comprise spatial positions in which thesynthetic aperture microphone has minimal audio reception sensitivity.

FIG. 10 is an illustration showing an exemplary localized primaryhot-spot 10-3 and a surrounding rejection region 10-8. Within theprimary hot-spot 10-3, the synthetic aperture microphone 10-2 can detectsound waves produced by a speaker 10-1. The location of the primaryhot-spot may be adjusted in accordance with particular conditions in anacoustic environment. In one embodiment, the position or orientation ofthe primary hot-spot may be modified under software control. This inturn could facilitate user-directed hot-spot positioning for optimizingaudio performance in different acoustic situations. FIG. 11 is anillustration showing exemplary primary hot-spot directivity, where thesynthetic aperture microphone 11-2 captures directionally-specificspeech energy from a user 11-1 within a primary hot-spot 11-3 that isoffset relative to that shown in FIG. 10. A rejection region 11-8 existsoutside the primary hot-spot 11-3 in a conventional manner.

The synthetic aperture microphone can additionally reject reflectedspeech energy that originated within the primary hot-spot and thatapproaches the microphone array 140.1, 140.2 from angles beyond thosethat span the primary hot-spot. FIG. 12 is an illustration showingexemplary reflected speech energy rejection. The synthetic aperturemicrophone 12-2 detects sound waves produced by a user 12-1 within aprimary hot-spot 12-3. The synthetic aperture microphone 12-2 rejectssound waves 12-5, 12-6 originating within the primary hot-spot 12-3 andreflected from nearby surfaces because the reflected sound waves arelikely to travel through one or more rejection regions 12-8 along theirreflection path.

The synthetic aperture microphone is further advantageous by virtue ofgood ambient acoustical noise rejection performance. FIG. 13 is anillustration showing exemplary ambient audio noise rejection, in which asynthetic aperture microphone 13-2 rejects conversational noise 13-4 andvarious forms of outside or environmental noise 13-5, 13-6, 13-7. Thenoise and noise reflections traveling towards the microphone array140.1, 140.2 enter a rejection region 13-8 through various directions,and hence are strongly attenuated via the synthetic aperturemicrophone's directional rejection behavior. This is in contrast to auser 13-1 within a primary hot-spot 13-3, who produces sound waves thatthe synthetic aperture microphone 13-2 readily detects with highsensitivity.

Referring also now to FIGS. 5 and 9, the synthetic aperture microphoneunit 330 outputs a mono microphone signal having a magnitude that mostdirectly corresponds to the amount of audio energy present within theset of hot-spots, and in particular the primary hot-spot. The syntheticaperture microphone output signal has little contribution from audioenergy entering from any rejection region directions. Those of ordinaryskill in the art will understand that multiple microphones can be usedto extract voice information from background noise that is in factlouder than the actual speech using adaptive cancellation techniquessuch as those described by Boll and Pulsipher in IEEE Transactions onAcoustics, Speech, and Signal Processing, Vol. ASSP-28, No. 6, December1980. This could be incorporated as a third operational mode for theaudio DSP, for supporting extreme noise environments as might be foundon public streets or repair depots, for example.

The volume control unit 350 is coupled to the left and right speakersignals, as are the SM/LS and SM/RS models 332, 334. The signal summingcircuit 336 is coupled to the output of the synthetic aperturemicrophone unit 340, as well as outputs of the SM/LS and SM/RS models332, 334, and delivers an echo-canceled mono synthetic aperturemicrophone signal to the A/V switch 106.

In one embodiment, the adaptive echo-canceled synthetic aperturemicrophone unit 105 comprises DSP hardware and/or software. The presentinvention can thus provide either adaptive echo-canceled stereomicrophone or adaptive echo-canceled mono synthetic aperture microphonecapabilities in response to user selection. In an exemplary embodiment,the adaptive echo-canceled synthetic aperture microphone unit 105 isimplemented in a DSP such as the Texas Instruments TMS320C54x processorreferenced above. Those skilled in the art will recognize that a singleDSP system can be configured to provide both the adaptive echo-canceledstereo and mono synthetic aperture microphone capabilities describedherein as distinct or integrated operating modes.

In the event that a user employs an earphone, headphone set, or AUXaudio devices in conjunction with the present invention, the syntheticaperture microphone unit 330 is placed in a bypassed, inactive, orquiescent state and the DSP and/or volume control unit 350 facilitateconventional normalization and adjustment of output signal amplitude, ina manner understood by those skilled in the art. Alternatively, separatenormalization and/or volume control circuitry could be provided toaccommodate the aforementioned devices.

Referring again to FIG. 5, the A/V switch 106 comprises conventionalanalog switching circuitry that is coupled to the preamplifier andbuffer unit 102, the audio signal conditioning unit 104, the analog A/VUTP transceiver 108, and the A/D-D/A converters 116 a. The A/V switch106 further maintains a coupling to the internal bus 115, therebyfacilitating processor control over A/V switch operation.

The A/V switch 106 routes incoming signals generated by the left andright microphones 140.1, 140.2 (or larger microphone array), the camera142, and/or any AUX A/V input devices to the analog A/V UTP transceiver108 or the A/D-D/A converters 116 a under the direction of a controlsignal received via the internal bus 115. Similarly, the A/V switch 106selects either the analog A/V UTP transceiver 108 or the A/D-D/Aconverters 116 a as a source for outgoing signals directed to the leftand right speakers 144.1, 144.2, the display device 146, and/or any AUXA/V output devices.

The analog A/V UTP transceiver 108 comprises a conventional analog A/Vtransceiver that provides a signal interface to a first set of UTP wiresthat carry analog A/V signals and which couple the analog A/V UTPtransceiver 108 to the pair splitter 110. The pair splitter 110 isfurther coupled to the first digital transceiver 111 via a second set ofUTP wires that carry digital A/V signals. The analog A/V UTP transceiver108 may be reconfigurable, supporting a range of analog 4-pair, 2-pair,or 1-pair signal transmission methodologies. The selection of anyparticular signal transmission methodology may be performed underprocessor control or by physical configuration switching. Similarly,distance compensation adjustments may be performed under processorcontrol or via physical switching, or alternatively through automaticcompensation techniques in a manner understood by those skilled in theart.

The first and second digital transceivers 111, 135 provide conventionaldigital interfaces to UTP wiring, and are coupled to the routing unit112 in the manner shown. The second digital transceiver 135 is furthercoupled to the companion computer port 136. The first and second digitaltransceivers 111, 135 may be implemented using portions of a standardNIC, as described below, or by other means. In addition to theaforementioned couplings, the routing unit 112 is coupled to the networkinterface unit 114. The routing unit 112 comprises conventional networkhub or mini-hub circuitry. In one embodiment, the routing unit 112performs hard-wired signal distribution and merge functions. In analternate embodiment, the routing unit 112 performs data packet deliverypath selection operations.

The network interface unit 114 comprises conventional network interfacecircuitry, for exchanging data with the internal bus 115 and datapackets with either the multimedia network 60 or a companion computer 50via the premises and companion computer network ports 138, 136 inaccordance with a conventional networking protocol. In one embodiment,the network interface unit 114 is implemented as at least one standardNIC. The NIC may typically include built-in data packet addressexamination or screening capabilities, and hence simplify the routingunit's function to one of communications distribution and mergefunctions in such an embodiment. These distribution and merge functionsserve to provide simultaneous signal or packet exchange among each ofthe premises network port 138, the NIC 114, and the companion computerport 136. One advantage of an embodiment employing a standard NIC isthat the NIC could be easily replaced or upgraded to accommodatetechnological evolution. This range of possibilities is further enhancedby the switching arrangement described below with reference to FIG. 15.Although not shown, it is again understood that should ISDN support bedeemed valuable, network connectors, interface electronics, inversemultiplexers, and Q.931 call control can be introduced through, forexample, connection to the internal bus 115 in a manner familiar tothose skilled in the art.

Taken together, the premises network port 138, the pair splitter 110,the analog A/V UTP transceiver 108, the digital transceiver 111, therouting unit 112, the network interface unit 114, and the companioncomputer port 136 form 1) a first multimedia network interface forhandling analog A/V signals; and 2) a second multimedia networkinterface for handling digital A/V and data networking signals. FIG. 14is a block diagram of a first embodiment of a first 400 and a second 410multimedia network interface provided by the present invention. Thefirst multimedia network interface 400 comprises the aforementionedfirst set of UTP wires plus the analog A/V UTP transceiver 108. Thefirst multimedia network interface 400 facilitates the exchange ofanalog A/V signals between the premises network port 138 and the analogA/V UTP transceiver 108. The second multimedia network interface 410comprises the second set of UTP wires, the digital transceiver 111, therouting unit 112, and the network interface unit 114, which are coupledin the manner shown. In some implementations, the digital transceiver135 may also be a NIC that may be either similar to or different from aNIC employed in the network interface unit 114. The second multimediainterface 410 facilitates the exchange of digital A/V and datanetworking signals between the premises network port 138, the networkinterface unit 114, and the companion computer port 136.

FIG. 15 is a block diagram of a second embodiment of first and secondmultimedia network interfaces provided by the present invention. Thefirst and second multimedia network interfaces are implemented via apassive switching arrangement and/or an active analog switching matrix420 that includes low-capacity, high-frequency analog protectiondevices. Such protection devices may comprise three-terminal,back-to-back diode arrangements, as employed in a Motorola BAV99LT1(Motorola, Inc., Schaumberg, Ill.). In this arrangement, the analogtransceiver 108 may support a number of 4, 2, and 1 pair formats, whichmay be dictated by the marketplace. Alternatively, the analogtransceiver 108 can be a replaceable module.

In the event that data networking is implemented via Gigabit Ethernet orother network protocol that conventionally consumes the entire physicalcapacity of an entire RJ-45 connector, the present invention may employan additional RJ-45 or other type of connector for carrying analog A/Vsignals.

Via the second multimedia network interface, the present inventionprovides internal data communication transmit, receive, and routingcapabilities. An external or companion computer 50 can therefore issuecontrol signals directed to the present invention in accordance withstandard data networking protocols. The second multimedia networkinterface can also provide “loop-through” signal routing between thepremises network port 138 and the companion computer port 136.Additionally, the data routing capabilities provided by the secondmultimedia network interface facilitate coupling to both existingbroadcast or switching hubs. The second multimedia network interfacealso supports the transfer of digital A/V streams. Thus, the secondmultimedia network interface cleanly separates data communicationsdirected to one or more companion computers 50, the multimedia network60, and the multimedia collaboration device 10.

Once again referring to FIG. 5, each of the A/V switch 106, the analogA/V UTP transceiver 108, the routing unit 112, the network interfaceunit 114, the A/V codec 116 b, the set of processors 118.1, 118.n, thememory 120, and the I/O interface 130 is coupled to the internal bus115. The A/V codec 116 b is further coupled to the A/D-D/A converters116 a, which are coupled to the A/V switch 106. It is noted that theA/D-D/A converters 116 a may include color-space conversion capabilitiesto transform between RGB and YUV or other advantageous color spaces.

The memory 120 comprises Random Access Memory (RAM) and Read-Only Memory(ROM), and stores operating system and application software 122, 124.Depending upon the nature of the processors 118.1, 118.n, the operatingsystem 122 could comprise a scaled-down, conventional, or enhancedversion of commercially-available operating system software, and/orspecial-purpose software. In an exemplary embodiment, the operatingsystem 122 comprises Windows CE (Microsoft Corporation, Redmond, Wash.)or another commercial product selected in accordance with the particularenvironment in which the present invention is employed. The applicationsoftware 124 may comprise programs for performing videoconferencing,messaging, publishing, broadcast reception, and media-on-demandoperations, and internet browsing using programs such as NetscapeNavigator (Netscape Communications Corporation, Mountain View, Calif.).Depending upon the nature of the processors 118.1, 118.n, the internetbrowser program could be a scaled down, conventional, or augmentedversion of a commercially-available browser.

The processors 118.1, 118.n manage communication with the networkinterface unit 114, and control the overall operation of the multimediacollaboration device 10 in accordance with control signals received viathe network interface unit 114. The processors 118.1, 118.n additionallyprovide graphics overlay capabilities, and may further provide internetbrowsing capabilities in conjunction with application software 124 aspreviously described. Relative to managing communication with thenetwork interface unit 114, the processors 118.1, 118.n may manageprotocol stacks and/or state machines. With regard to controlling theoverall operation of the multimedia collaboration device 10, theprocessors 118.1, 118.n issue control signals to the A/V switch 106 andexecute application software resident within the memory 120. Thegraphics overlay capabilities facilitate the placement of fonts,cursors, and/or graphics over video present upon the display device 146.With sufficient processing power, the present invention can serve as astand-alone, real-time video-capable internet appliance.

As described above, the A/D-D/A converters 116 a may compriseconventional circuitry to perform color-space conversion operations inaddition to analog-to-digital and digital-to-analog signal conversion.The A/V codec 116 b comprises conventional A/V signal encoding anddecoding circuitry, and provides the present invention with compressionand decompression capabilities. Together these enable the presentinvention to encode and decode A/V streams without loading down acompanion computer's processing and networking power. Either of thefirst or second multimedia network interfaces described above can routedigital A/V signals to the A/V codec 116 b, while routing non-A/Vsignals to the companion computer 50. The present invention's ability toencode and decode A/V signals independent of a companion or externalcomputer is particularly advantageous in situations in which videosignal encoding and decoding must occur simultaneously, such as in 2-wayteleconferencing or network-based video editing applications. Thepresent invention may support network-based video editing applicationsbased upon a high bandwidth near-zero-latency compression approach,which can be implemented, for example, through JPEG or waveletcompression operations; or an interim compression approach.

In one embodiment, the A/V codec 116 b comprises a chip or chipset. Inanother embodiment, the A/V codec 116 b comprises a processor 118.kcapable of performing compression and decompression operations. In moreadvanced implementations, the A/V codec 116 b could comprise a singleprocessor 118.m capable of performing user interface functions inaddition to A/V compression and decompression operations. Such animplementation could also provide an Application Program Interface (API)in conjunction with operating system software 122. In an exemplaryembodiment of such an implementation, the A/V codec 116 b may comprise aNUON processor (VM Labs, Mountain View, Calif.).

4.4 Additional Embodiments

FIG. 6 is a block diagram of a second embodiment of a multimediacollaboration device 20, which provides primary support for analog audioI/O and digital visual I/O, and further supports analog and digitalauxiliary A/V I/O, plus networked digital streaming. Relative to FIG. 5,like reference numbers designate like elements.

The second embodiment of the multimedia collaboration device 20 includesa digital camera 152, a digital display device 154, a digital AUX A/Vinterface 156, and a stream selector 158. The digital camera 152 and thedigital display device 154 respectively capture and display images in aconventional manner. The digital AUX A/V interface 156 facilitatesbidirectional coupling to auxiliary digital A/V devices, such as anexternal computer, a digital VCR, or Digital Versatile Disk (DVD)player. Each of the digital camera 152, the digital display device 154,and the digital AUX A/V interface 156 is coupled to the stream selector158, which is coupled to the A/V codec 116 b.

The stream selector 158 comprises conventional circuitry thatselectively routes digital streams between the A/V codec 116 b and thedigital camera 152, the digital display device 154, and the digital AUXA/V interface 156. The stream selector 158 may route incoming digitalimage streams received from either of the digital camera 152 or thedigital AUX A/V interface 156 to the A/V codec 116 b. In one embodiment,the stream selector 158 may be capable of multiplexing between these twoincoming digital stream sources. Undersampling may also be used tofacilitate the compositing of multiple video images. Relative tooutgoing digital image streams, the stream selector 158 may route suchstreams to either or both of the digital display device 154 and digitalAUX A/V interface 156, where such routing may occur in a simultaneous ormultiplexed manner. The stream selector 158 additionally facilitates theexchange of digital audio streams between the A/V codec 116 b and thedigital AUX A/V interface 156.

The A/V codec 116 b and the A/D-D/A converters 116 a together facilitatethe conversion of digital A/V signals associated with the digital camera152, the digital display device 154, and/or auxiliary digital A/Vdevices into analog A/V signals. The A/V switch 106 facilitates exchangeof these analog A/V signals with AUX A/V devices and/or the premisesnetwork port 138.

Because the A/V codec 116 b is also coupled to the internal bus 115 andhence to the network interface unit 114, digital A/V signals capturedfrom the digital camera 152 or directed to the digital display 154 orreceived from the digital AUX A/V interface 156 may be packetized andexchanged via the premises network port 138 and/or the companioncomputer port 136.

FIG. 7 is a block diagram of a third embodiment of a multimediacollaboration device 30, which provides primary support for analog audioI/O and digital visual I/O, support for digital auxiliary A/V I/O, andsupport for networked digital streaming. Relative to FIGS. 5 and 6, likereference numbers designate like elements.

The third embodiment of the multimedia collaboration device 30 includesa digital camera 152, a digital display device 154, a digital AUX A/Vinterface 156, and a stream selector 158 in the manner described above.Analog audio signals associated with the microphones 140.1, 140.2 andspeakers 144.1, 144.2 are routed through the A/D-D/A converters 116 aand A/V codec unit 116 b. Thus, the third embodiment of the presentinvention manages digital A/V streams, and may exchange such streamswith the multimedia network 60 and/or a companion computer 50. The thirdembodiment of the multimedia collaboration device 30 does not transmitanalog A/V signals over the multimedia network 60, and hence the analogswitching unit 106, the analog A/V UTP transceiver 108, and the pairsplitter 110 described above relative to the first and second multimediacollaboration device embodiments are not required.

4.5 Camera and Display Device Integration

As previously indicated, placement of the camera 142 in close proximityto the display device 146 aids in maintaining good user eye-contact witha displayed image, thereby closely approximating natural face-to-facecommunication in videoconferencing situations. Essentially perfecteye-contact can be achieved by integrating a large-area photosensorarray with a large-area array of emissive or transmissive devices thatform the basis for display device pixels.

Multiple photosensor and display element integration techniques exist.In general, the formation of an image using a photosensor arraynecessitates the use of optical elements in conjunction with photosensorelements. Photosensor and display element integration techniques aredescribed in detail hereafter, followed by image formationconsiderations relative to integrated photosensor/display elementarrays.

4.6 Display Pixel and Photosensor Element Interleaving

One way of integrating photosensor elements with emissive ortransmissive display elements is via element interleaving. FIG. 16 is anillustration showing a first photosensor and display elementinterleaving technique, in which display elements 510 and photosensorelements 520 populate a viewing screen 502 in an alternating manner.Each display element 510 generates or transmits light corresponding to aparticular color or set of colors. Similarly, each photosensor element520 detects light corresponding to a particular color. As described indetail below, display elements 510 and photosensor elements 520 operatein a temporally and/or spatially separated manner relative to each otherto ensure that image capture is essentially unaffected by image display.

Display and photosensor elements 510, 520 corresponding to a particularcolor are interleaved in accordance with a color distribution scheme.FIG. 17 is an illustration of an exemplary photosensor element color anddisplay element color distribution scheme. In FIG. 17, display elements510 corresponding to the colors red, green, and blue are identified viathe uppercase letters R, G, and B, respectively. Photosensor elements520 corresponding to red, green, and blue are respectively identified bythe lowercase letters r, g, and b. Display elements 510 corresponding toa particular color are offset relative to each other, and interleavedwith display and photosensor elements 510, 520 corresponding to othercolors. Similarly, photosensor elements 520 corresponding to aparticular color are offset relative to each other, and interleaved withdisplay and photosensor elements 510, 520 corresponding to other colors.Those skilled in the art will recognize that a variety of photosensorand display element color distribution schemes are possible.

The presence of photosensor elements 520 interleaved with displayelements 510 reduces image resolution, and increases pixel pitch (i.e.,the spacing between pixels). To minimize the effect that the photosensorelements 520 have upon the appearance of a displayed image, photosensorelements 520 having or consuming a smaller area than the displayelements 510 are employed. Furthermore, various display and photosensorelement layout geometries may be used to produce an interleaving patternthat closely approximates display element pitch found in conventionaldisplay devices. FIG. 18 is an illustration of a second photosensor anddisplay element interleaving technique, in which photosensor and displayelement geometries and size differentials aid in minimizing pixel pitchand maximizing displayed image resolution. Since a viewer's eye willintegrate or average the light output by groups of display elements 510,interleaving techniques of the type shown in FIG. 18 ensure that theviewer will perceive a high-quality image. Those skilled in the art willunderstand that various microoptic structures or elements, such asmicrolenses, could be employed in the nonluminent spaces between displayelements 510 and/or photosensor elements 520 to reduce or minimize aviewer's perception of nonluminent areas in a displayed image. Suchmicrooptic structures are elaborated upon below.

The display elements 510 referred to herein may comprise essentially anytype of conventional light emitting or transmitting device, such as aLight Emitting Diode (LED) or Liquid Crystal Display (LCD) pixelelement. Similarly, the photosensor elements 520 may compriseessentially any type of conventional light sensing or detecting device.For example, the photosensor elements 520 could comprise photodiodes,such as Schottky or p-i-n photodiodes; phototransistors; capacitive orcharge-coupled devices (CCDs); charge modulated devices (CMDs); or othertypes of light-sensitive devices. The photosensor elements 520 could befabricated, for example, using standard semiconductor processingtechniques employed during the manufacture of flat panel displays.

In a typical display device, a single display element 510 is used tooutput light of a particular color. Display elements 510 based uponorganic electroluminescence are capable of simultaneously generatinglight comprising multiple wavelengths in the visible spectrum, and formthe basis for full-color LED arrays. In particular, a single StackedOrganic Light Emitting Diode (SOLED) pixel element can produce red,green, and blue light. The intensity of each color is independentlytunable, as is each color's mean wavelength. Thus, a single SOLED canform a full-color pixel. As an alternative to organic electroluminescentmaterials, the present invention may employ other full-color transparentor semitransparent luminescent materials, such as light-emitting and/orlight-responsive polymer films.

FIG. 19 is a cross-sectional view showing a full-color pixel arrayintegrated with a photosensor element array upon a common substrate 702such as glass or plastic. As an example, a SOLED 710 is considered asthe full-color pixel technology in the discussion that follows. Thoseskilled in the art will understand that the concepts described hereincan be applied to other full-color pixel technologies. Each SOLED 710comprises a first, second, and third semitransparent electrode 712, 714,716; a first, second, and third organic electroluminescent layer 722,724, 726; and a reflecting contact layer 730, in a manner understood bythose skilled in the art. Each electroluminescent layer 722, 724, 726emits light in a particular wavelength range in response to an appliedelectric field. For example, the first, second, and third organicelectroluminescent layers 722, 724, 726 could respectively output blue,green, and red light.

A color filter 750, an optional microoptic structure 760, and aphotosensor element 520 form a color-specific photosensor element 770that is fabricated adjacent to each SOLED 710. The microoptic 760 maycomprise one or more microlenses, apertures, and/or other types ofplanar optic structures, and serves to focus incoming light onto thephotosensor element 520 to aid image formation in the manner describedbelow. The microoptic structure 760 may be formed through theapplication of conventional microlens or planar optic fabricationtechniques during photosensor element fabrication steps. For example,the microoptic structure 760 may be formed by depositing aselectively-doped dielectric or dielectric stack prior to or duringphotosensor element fabrication, in a manner well understood by thoseskilled in the art.

The color-specific photosensor element 770 may also include one or moreantireflection layers, which are deposited in a conventional manner.Additionally, one or more types of passivation or isolation materials,such as Silicon Dioxide, Silicon Nitride, Polyimide, or spin-on-glassmay be deposited in between each SOLED 710 and color-specificphotosensor element 770 in a manner understood by those skilled in theart.

Each color-specific photosensor element 770 detects light characterizedby a specific wavelength interval. Thus, while any given SOLED 710 maysimultaneously output red, green, and/or blue light, separatecolor-specific photosensor elements 770 are used to individually detectred, green, and blue light. Because each SOLED 710 forms a full-colorpixel, integration of a SOLED array with a photosensor array in themanner shown in FIG. 19 is particularly advantageous relative toproviding a high-resolution display having image capture capabilities.

4.7 Display and Photosensor Element Stacking

a) Integrated SOLED/Photosensor Element

A full-color pixel element such as a SOLED 710 and a color-specificphotosensor element 770 can be integrated together, such that theincorporation of a photosensor element array into a display elementarray can be accomplished essentially without a resolution or pixelpitch penalty. FIG. 20 is a cross-sectional view showing an integratedfull-color pixel/photosensor element 800, which may form the basis of anintegrated display element/photosensor element array. For purpose ofexample, the full-color pixel element is considered to be a SOLED 810 inthe description that follows. Those skilled in the art will understandthat other types of full-color pixel technologies could be used toproduce the integrated full-color pixel/photosensor element 800described hereafter.

Relative to FIG. 19, like reference numbers designate like elements. Thefull-color pixel/photosensor element 800 comprises a SOLED 810 having acolor-specific photosensor element 770 fabricated thereupon. Thefull-color pixel/photosensor element 800 is fabricated upon a substrate702 such as glass. The SOLED 810 comprises a first, a second, a third,and a fourth semitransparent electrode 712, 714, 716, 812; a first,second, and third organic electroluminescent layer 722, 724, 726; and apatterned reflecting contact layer 830.

With the exception of the fourth semitransparent electrode 812 and thepatterned reflecting contact layer 830, the SOLED 810 shown in FIG. 20is essentially the same as that depicted in FIG. 19. The fourthsemitransparent electrode 812 serves as one of the electrodes for thephotosensor element 520 within the color-specific photosensor element770, in a manner readily understood by those skilled in the art.Deposition of the fourth semitransparent electrode 812 may not berequired under the patterned reflecting contact layer 830, and as suchthe SOLED 810 and color-specific photosensor element 770 may not share acommon electrical interface layer. The patterned reflecting contactlayer 830 comprises conventional contact materials or metals that havebeen patterned to include a gap or opening.

The color-specific photosensor element 770 is fabricated on top of thefourth semitransparent electrode 812, in the opening defined in thepatterned reflecting contact layer 830. The color-specific photosensorelement 770 thus detects light that has been transmitted through thesubstrate 702 and each of the first through fourth semitransparentelectrodes 712, 714, 716, 812. Those skilled in the art will understandthat the location of the opening defined in the patterned reflectingcontact layer 830, and hence the location of the color-specificphotosensor element 770 upon the SOLED 810, may vary among adjacentfull-color pixel/photosensor elements to ensure that the human observerperceives a high-quality displayed image. The SOLED 810 and thecolor-specific photosensor element 770 may operate in atemporally-separated manner to ensure that image capture is essentiallyunaffected by image display, as further elaborated upon below.

b) Stacked Full-Color Emitter/Full-Color Detector Structures

A full-color pixel element, such as a stacked organic electroluminescent(SOE) structure, may also be used to detect light. Thus, a singlestructure based upon full-color materials technology may be used forboth RGB light emission and RGB light detection, thereby advantageouslyfacilitating the integration of a photosensor element array and adisplay element array while maintaining small pixel pitch and high imageresolution.

FIG. 21 is a cross-sectional view of a first full-color emitter/detector900. In the description that follows, the first full-coloremitter/detector 900 is considered to be an SOE-based device. Thoseskilled in the art will recognize that other full-color technologiescould be employed to produce the first full-color emitter/detector 900in alternate embodiments.

Relative to FIGS. 19 and 20, like reference numbers designate likeelements. The first full-color emitter/detector 900 is fabricated upon asubstrate 702 such as glass, and comprises first through seventhsemitransparent electrodes 712, 714, 716, 812, 912, 914, 916; firstthrough sixth organic electroluminescent layers 722, 724, 726, 922, 924,926; an optional microoptic structure 920; and a reflecting contactlayer 730.

In the first full-color emitter/detector 900, the first through thirdorganic electroluminescent layers 722, 724, 726 serve as RGB lightemitters controlled by voltages applied to the first through fourthsemitransparent electrodes 712, 714, 716, 812, and thus form a SOLED902. The microoptic structure 920 comprises one or more microlenses,apertures, and/or other planar microoptic structures that focus incominglight into the fourth, fifth, and sixth organic electroluminescentlayers 922, 924, 926, which in turn produces or induces pairwise voltagedifferences across the fifth, sixth, and seventh semitransparentelectrodes 912, 914, 916 and the reflecting contact layer 730. Themicrooptic structure 920, the fourth through sixth organicelectro-luminescent layers 922, 924, 926, the fifth through seventhsemitransparent electrodes 912, 914, 916, and the reflecting contactlayer 730 therefore form a first SOE photosensor 904 for detecting RGBlight.

Light emitted by the SOLED 902 may travel through the substrate 702toward a viewer, or through the first SOE photosensor 904, where it isreflected back toward the substrate 702 by the reflecting contact layer730. The first SOE photosensor 904 detects incoming light that hastraveled through the substrate 702 and the SOLED 902. As described indetail below, SOLED light emission and SOE photosensor light detectionmay occur in a temporally and/or spatially separated manner, such thatimage capture is essentially unaffected by image display.

Those skilled in the art will recognize that the SOLED 902 and the firstSOE photosensor 904 may be able to share a single semitransparentelectrode at their interface in an alternate embodiment (i.e., the firstfull-color emitter/detector 900 may be fabricated without one of thefourth or fifth semitransparent electrodes 812, 912) since SOLED and SOEphotosensor operation within a single first full-color emitter/detector900 may be temporally separated). Those skilled in the art will alsounderstand that in addition to the layers described above, the firstfull-color emitter/detector 900 may include additional microoptic layersand/or one or more antireflective layers. Those skilled in the art willfurther recognize that in an alternate embodiment, the first full-coloremitter/detector 900 could be fabricated such that the first SOEphotosensor 904 resides in contact with the substrate 702, and the SOLED902 resides on top of the first SOE photosensor 904. In such anembodiment, the reflecting contact layer 730 would be incorporated intothe SOLED 902. Those skilled in the art will also recognize that eitheror both of the SOLED 902 and the first SOE photosensor 904 could beimplemented using other types of transparent or semitransparentfull-color device and/or materials technologies in alternateembodiments.

FIG. 22 is a cross-sectional view of a second full-coloremitter/detector 1000. For ease of understanding, the second full-coloremitter/detector is considered to be based upon SOE technology in thefollowing description. Those skilled in the art will recognize thatother full-color materials technologies could be employed to produce thesecond full-color emitter/detector 1000 in alternate embodiments.

Relative to FIG. 20, like reference numbers designate like elements. Thesecond full-color emitter/detector 1000 is fabricated upon a substrate702 such as glass, and comprises a first through fifth semitransparentelectrode 712, 714, 716, 812, 1012; a first through sixth organicelectroluminescent layer 722, 724, 726, 1022, 1024, 1026; an optionalmicrooptic structure 1020; a first, a second, and a third reflectingcontact layer 1032, 1034, 1036; and a first and a second boundarystructure 1042, 1044.

The first through third organic electroluminescent layers 722, 724, 726,in conjunction with the first through fourth semitransparent electrodes712, 714, 716, 812, form a SOLED 902 in a manner analogous to thatdescribed above with reference to FIG. 21. The microoptic structure1020, the first through third organic electroluminescent layers 1022,1024, 1026, the reflecting contact layers 1032, 1034, 1036, and thefirst and second boundary structures 1042, 1044 form a second SOEphotosensor 1004.

Taken together, the fourth, fifth, and sixth organic electroluminescentlayers 1022, 1024, 1026 and the boundary structures 1042, 1042 span anarea essentially equal to that of any semitransparent electrode 712,714, 716, 812, 1012. The first boundary structure 1042 separates thefourth and fifth organic electroluminescent layers 1022, 1024.Similarly, the second boundary structure 1044 separates the fifth andsixth organic electroluminescent layers 1024, 1026. The first, second,and third reflecting contact layers 1032, 1034, 1036 respectively resideupon or atop the fourth, fifth, and sixth organic electroluminescentlayers 1022, 1024, 1026.

The microoptic structure 1020 may comprise one or more microlenses,apertures, and/or other planar microoptic structures that focus incominglight into the fourth, fifth, and sixth organic electroluminescentlayers 1022, 1024, 1026. The fourth organic electroluminescent layer1022 detects incoming photons having a wavelength range associated witha particular color, for example, red. The presence of such photons inthe fourth organic electroluminescent layer produces or induces avoltage difference between the fourth semitransparent electrode 1012 andthe first reflecting contact layer 1032. Similarly, the fifth and sixthorganic electroluminescent layers 1024, 1026 each detect incoming lightcorresponding to a particular wavelength range, for example, green andblue, respectively. The presence of blue and green light respectivelyinduces a voltage difference between the second and third reflectingcontact layers 1034, 1036 and the fourth semitransparent electrode 1012.

Those skilled in the art will recognize that the thickness of each ofthe fourth, fifth, and sixth organic electroluminescent layers 1022,1024, 1026 may be varied in accordance with the particular wavelengthrange that each such layer is to detect. Those skilled in the art willadditionally recognize that the microoptic structure 1020 may befabricated such that its characteristics vary laterally from one organicelectroluminescent layer 1022, 1024, 1026 to another, and that one ormore antireflection layers may be incorporated into the secondfull-color emitter/detector 1000. Moreover, the SOLED 902 and the secondSOE photosensor 1004 may be able to share a single semitransparentelectrode at their interface a manner analogous to that described aboverelative to the first SOE photosensor 904. Finally, those skilled in theart will recognize that either or both of the SOLED 902 and the secondSOE photosensor 1004 could be implemented using other types oftransparent or semitransparent full-color technologies in alternateembodiments.

4.8 Other Integrated Emitter/Detector Structures

As indicated above, a light detecting element may be similar, nearly, oressentially identical in structure and/or composition to a lightemitting element. Because any given emitter/detector structure may beused for light emission during one time interval and light detectionduring another time interval as described below, a single light emittingstructure may also be used for light detection.

FIG. 23 is a cross-sectional diagram of a third full-coloremitter/detector 1100. For ease of understanding, the third full-coloremitter/detector is described hereafter in the context of SOEtechnology. Those skilled in the art will understand that otherfull-color materials and/or technologies could be employed to producethe third full-color emitter/detector 1100 in alternate embodiments.

Relative to FIG. 19, like reference numbers designate like elements. Thethird full-color emitter/detector 1100 is fabricated upon a substrate702 such as glass or plastic. The third full-color emitter/detector 1100comprises a SOLED 710 having a first through a third semitransparentelectrode 712, 714, 716; a first, a second, and a third organicelectroluminescent layer 722, 724, 726; a reflecting top contact layer730. The third full-color emitter/detector 1000 may additionally includea microoptic layer 1120. During a first time interval, the SOLED 710 mayoperate in a light emitting mode in a conventional manner. During asecond time interval, the SOLED 710, in conjunction with the microopticlayer 1120, operates as a photosensor to detect incoming light in amanner analogous to that described above relative to the SOEphotosensors 904.

The microoptic layer 1120 may comprise a microlens and/or other type ofplanar optic structure, and may be fabricated such that differentportions of the microoptic layer 1120 affect light in different manners.This in turn could aid in providing particular light detectionresponsivity while minimally affecting the manner in which light emittedby the third full-color emitter detector 1100 will be perceived by ahuman eye.

FIG. 24 is a top-view of an exemplary microoptic layer 1120 havingdifferent optical regions 1190, 1192 defined therein. A first opticalregion 1190 may allow light to pass in an essentially unaffected manner.A second optical region 1192 serves as a focusing element that producesa desired spatial or modal light intensity pattern within the thirdfull-color emitter/detector. As the second optical region 1192 occupiesa smaller area than the first optical region 1190, its affect upon humanperception of light emitted by the third full-color emitter/detector maybe small or minimal. Those skilled in the art will understand that thelocation of the second optical region 1192 may vary among adjacent thirdfull-color emitter/detectors 1100, to further enhance the quality of adisplayed image seen by a human eye.

In an alternate embodiment, the microoptic layer 1120 could includeadditional optical regions. For example, one or more portions of thefirst optical region 1190 could be designed or fabricated to compensatefor any effects the second optical region 1192 has upon human perceptionof light emitted by the third full-color emitter/detector 1100. Asanother example, the second optical region 1192 could be replaced oraugmented with other, possibly smaller, optical regions distributedacross the plane of the microoptic layer 1120 to further optimize lightdetection and emission characteristics.

4.9 Image Formation

A simple or compound lens is conventionally used to focus an image ontoan array of photosensors. FIG. 25 illustrates a simple or compound lens600 that receives or collects light 602 reflected or emanating from anobject 604, and focuses such light onto a photosensor element array 606.

Relative to a single array that integrates both display and photosensorelements 510, 520, the use of a conventional simple or compound lenswould adversely affect the characteristics of the displayed image. Tofacilitate image detection in such an integrated array, photosensorelements 520 may incorporate microoptic structures and/or apertures, asdescribed above, on an individual basis. Each aperture and/or microopticstructure focuses light received from a small portion of an object ontoa photosensor element 520. As depicted in FIG. 25, sets ofmicrooptic-equipped photosensor elements 520 within a photosensor array620 receive light 622, 624 emanating from different parts of an object626. Those skilled in the art will recognize that the present inventioncould employ microoptic structures or elements that focus light ontomultiple photosensor elements 520 in alternate embodiments, where suchmicrooptic elements may be incorporated onto separates substrates.Signals output by the microoptic-equipped photosensor elements 520 aretransferred to an image processing unit 628 for further processing, asdescribed in detail below.

Conventional display devices comprise multiple rows or lines of displayelements 510, and produce a displayed image on a line-by-line basis.Similarly, conventional photosensor arrays comprise multiple rows ofphotosensor elements 520, which may be scanned on a line-by-line basisduring image capture operations. The integrated displayelement/photosensor element arrays considered herein may also 1) producea displayed image by activating display elements 510 on a line-by-linebasis; and 2) capture light received from an object by detectingphotosensor element output signals on a line-by-line basis.

In one embodiment, the present invention includes a display controlcircuit for performing display line scans that produce a displayed imageon a line-by-line basis, and a capture control circuit for performingphotosensor line scans that read photosensor element output signals on aline-by-line basis. Each of the display and capture control circuitsinclude conventional clocking, address decoding, multiplexing, andregister circuitry. In order to ensure that image capture is essentiallyunaffected by image display (i.e., to prevent light emitted ortransmitted by display elements 510 from affecting incoming lightdetection by adjacent photosensor elements 520), the display line scansand photosensor line scans may be temporally and/or physically separatedrelative to each other. This separation may be controlled viaconventional clocking and/or multiplexing circuitry.

In one embodiment, photosensor line scans are initiated after a displayline scan has generated fifty percent of an image (i.e., after fiftypercent of the display element lines have been activated during a singlefull-screen scan cycle), such that the photosensor line scan trails thedisplay line scan by a number of display element rows equal to one-halfof the total number of display element rows present in the integrateddisplay element/photosensor element array. More generally, the captureline scan could trail the display line scan by a particular timeinterval or a given number of completed display line scans.

In another embodiment, one-half of the display element lines define afirst display field, and one-half of the display element lines define asecond display field, in a manner well understood by those skilled inthe art. Similarly, one-half of the photosensor element lines define afirst photosensor field, and the remaining photosensor element linesdefine a second photosensor field. The first display field and either ofthe first or second photosensor fields may be scanned eithersimultaneously or in a time-separated manner, after which the seconddisplay field and the remaining photosensor field may be scanned in ananalogous manner. Those skilled in the art will recognize that thedisplay and photosensor field scanning can be performed in a manner thatsupports odd and even field scanning as defined for NTSC and PALtelevision standards.

In yet another embodiment, a single full-screen display line scan cycleis completed, after which a single full-screen photosensor line scancycle is completed, after which subsequent full-screen display line andphotosensor line scans are separately performed in a sequential manner.

The set of photosensor element output signals received during any givenphotosensor line scan are transferred to an image processing unit 628.The image processing unit 628 comprises signal processing circuitry,such as a DSP, that performs conventional digital image processingoperations such as two-dimensional overlap deconvolution, decimation,interpolation, and/or other operations upon the signals generated duringeach photosensor line scan. Those skilled in the art will understandthat the number and types of digital image processing operationsperformed upon the signals generated during each photosensor line scanmay be dependent upon the properties of any microoptic structuresassociated with each photosensor element 520. Those skilled in the artwill further understand that signal conditioning circuitry mayadditionally be present to amplify photosensor element signals oreliminate noise associated therewith. Such signal conditioningcircuitry, or a portion thereof, may be integrated with each photosensorelement 520.

The image processing unit 628 forms a conventional final output imagearray using signal processing methods, and outputs image array signalsto a buffer or memory, after which such signals may be compressed andincorporated into data packets and/or converted into analog videosignals for subsequent transmission, where the compression and/orconversion may occur in conjunction with associated audio signals.

The signal processing algorithms employed in image formation aredetermined by the nature of any microoptic elements employed inconjunction with the photosensor elements 520. Such algorithms mayperform deconvolution, edge-effect handling, decimation, and/orinterpolation operations in a manner understood by those skilled in theart.

For example, if the microoptic elements amount to tiny apertures thatlimit detector pixel source light to non-overlapping segments in theprincipal area of view, the signal processing amounts to aggregating thepixels into an array and potentially performing interpolation and/ordecimation operations to match the resolution of the pixel detectorarray to that of the final desired image.

As detection pixels overlap by increasing amounts, the applied signalprocessing operations can advantageously sharpen the image bydeconvolving the impulse response of the pixel overlap function.Depending upon the microoptic arrangement employed, which may bedictated by device cost and fabrication yield or reliability, theoverlap impulse response takes on varying characteristics, affecting thealgorithm the image processing unit 628 is required to perform. Ingeneral, the deconvolution can be handled as either a set oftwo-dimensional iterated difference equations, which are readilyaddressed by standard numerical methods associated with the approximatesolution of differential equations, or through conversions to thefrequency domain and appropriate division operations. Further, if theoverlap function is highly localized, which can be a typical situation,the difference equations can be accurately approximated by neglectinghigher-order terms, which greatly simplifies the resulting operations.This is in contrast to frequency domain techniques for this case, aslocalization in the impulse response implies immense nonlocalization inthe transform domain. However, should the overlap impulse responseitself be far less localized, frequency domain deconvolution methods maybe advantageous. Care must be taken in limiting the division to relevantareas when there are zeros in the frequency-domain representation of theoverlap impulse response (transfer function).

Edge effects at the boundaries of the pixel detector array can behandled by various methods, but if the overlap impulse response is keptlocalized by apertures and/or other microoptic elements, thenundesirable edge effects in the final image formation (that may resultfrom “brute-force” treatment of the edges) quickly vanish within a fewpixels from the boundary of the final formed image. Cropping can then beemployed to avoid such edge effect altogether. Thus, by creating aslightly-oversized pre-final image formation array and eliminating edgeeffect by cropping, a final image array of desired resolution having noedge effects induced by overlap impulse responses can be readilyproduced.

It is known to those skilled in the art that in general, apertureeffects invoked by actual apertures and/or microoptic elements cancreate diffraction patterns or spatial intensity modes in the lighttransmitted through the optical structure. Such optical structures maybe designed to enhance or eliminate particular modes or diffractioneffects, in a manner readily understood by those skilled in the art.

While the teachings presented above have been described in relation to adisplay device having a camera or image capture capabilities integratedtherein or thereupon, the above teachings relating to 1) variousphotosensor element, microoptic and/or apertured structures; and 2)image processing requirements for creating an array of image signalsthat correspond to a captured image can be applied to effectively createa camera disposed or integrated upon any one of a wide variety ofsurfaces or substrates, including glass, plastic, partially-silveredmirrors, or other materials. Photosensor elements 520 disposed upon suchsubstrates may be organized or distributed in a manner similar to thatshown above with reference to FIGS. 16, 17, and 18, with the exceptionthat display elements 510 shown in those figures may not be present.

The principles of the present invention have been discussed herein withreference to certain embodiments thereof. Study of the principlesdisclosed herein will render obvious to those having ordinary skill inthe art certain modifications thereto. The principles of the presentinvention specifically contemplate all such modifications.

The invention claimed is:
 1. A display device having image capturecapabilities for at least one distant image source, the display devicecomprising: a plurality of display elements; a plurality of photosensorelements, the plurality of photosensor elements interleaved with theplurality of display elements in an essentially planar arrangement, suchthat the essentially planer arrangement can function as a display forthe display device and to perform image capture simultaneously; whereinthe plurality of photosensor elements is configured to receive lightfrom a distant non-adjacent image source separated from the displaydevice by empty space, and wherein the photosensor elements createoutput signals responsive to the received light from a distantnon-adjacent image source for presentation to signal processing arrangedfor image formation to form a synthetic image, the synthetic imageresponsive to the light received from the distant non-adjacent imagesource.
 2. The display device of claim 1, wherein at least onemicrooptic structure is associated with a set of photosensor elements.3. The display device of claim 2, wherein a dedicated microopticstructure is associated with each photosensor element within the set ofphotosensor elements.
 4. A system for image formation comprising: a setof photosensor elements, each configured to generate a photosensoroutput signal in response to light received thereon from a distantnon-adjacent object; a plurality of display element interleaved with thephotosensor element in an essentially planar arrangement such that theessentially planer arrangement can function as a display and to performimage capture simultaneously, the display elements being configured toemit light representative of a respective display signal; a plurality ofmicro-optic elements, each of which is configured to direct light from aportion of the distant non-adjacent object separated from themicro-optic elements by empty space onto a respective subset of the setof photosensor elements; and at least one signal processor configuredto: receive photosensor output signals from the set of photosensorelements, perform an image formation operation on the receivedphotosensor output signals to create first image data, perform a digitalprocessing operation on the first image data to produce second imagedata, the digital processing operation including at least one of an edgeeffect handling operation to remove edge effects, a sharpening operationand an aggregation operation, and provide output image data based on thesecond image data, the output image data representing a portion of animage of the distant non-adjacent object.
 5. The system of claim 4,wherein the photosensor elements are arranged in a first array and themicro-optic elements are arranged in a second array.
 6. The system ofclaim 5, wherein the image formation operation includes a deconvolutionoperation based on an overlap impulse response determined by propertiesof respective micro-optic elements associated with each of the set ofphotosensor elements.
 7. The system of claim 6 wherein the deconvolutionoperation comprises a two-dimensional overlap deconvolution operation.8. The system of claim 4, wherein the sharpening operation comprisesremoving effects in the first image data due to overlap of thephotosensor elements.
 9. The system of claim 4, wherein the micro-opticelements are fabricated as a component of respective photosensorelements.
 10. The system of claim 4, wherein the micro-optic elementsare fabricated using at least one of a micro lens fabrication techniqueand a planar optic fabrication technique.
 11. The system of claim 4,wherein the at least one signal processor is further configured toperform at least one of a decimation operation and an interpolationoperation on at least one of the photosensor output signals and thefirst image data.
 12. The system of claim 4, wherein the edge effectsare caused by a first set of photosensor elements positioned within aperipheral region of the photosensor array.
 13. The system of claim 4,wherein the edge effect handling operation to remove edge effectscomprises a crop operation to remove from subsequent processing portionsof the first image data.
 14. The system of claim 4, wherein themicro-optic elements are configured in a configuration to reduce edgeeffects in the output image data.
 15. The system of claim 4, wherein themicro-optic elements are configured to have a localized overlap impulseresponse.
 16. The system of claim 4, wherein the aggregation operationcomprises the aggregation of processed versions of the first image data.17. A system for image formation comprising: a plurality of photosensorelements arranged in a first array; a plurality of display elementsinterleaved with the photosensor elements in an essentially planerarrangement such that the essentially planar arrangement can function asa display and to perform image capture simultaneously, the displayelements being configured to emit light representative of a respectivedisplay signal; a plurality of micro-optic elements, each of themicro-optic elements configured to direct received light from a portionof a distant non-adjacent object separated from the plurality ofmicro-optic elements by empty space, onto a respective subset of thephotosensor elements, each of the photosensor elements configured togenerate a photosensor output signal in response to received light; acapture control circuit configured to scan the output signals generatedby the photosensor elements on a line-by-line basis; and an imageprocessing unit configured to: receive from the capture control circuitthe photosensor output signals generated during a respective photosensorline scan; perform at least one digital image processing operation onthe photosensor output signals generated during the photosensor linescan; and based on the signals generated by the digital image processingoperation, provide respective output image signals, each of whichrepresents a respective portion of an image of the distant non-adjacentobject, wherein each of the respective image signals are responsive toreceive light from the distant nonadjacent image source.
 18. The systemof claim 17, further comprising: signal conditioning circuitryconfigured to perform at least one signal conditioning operationincluding at least one of amplifying the photosensor output signals andreducing noise in the output signals generated by the photosensorelements.
 19. The system of claim 18, wherein the signal conditioningcircuitry is integrated with the plurality of photosensor elements. 20.A method for image formation comprising: using a plurality ofmicro-optic elements to direct light received from a portion of adistant non-adjacent object separated from the plurality of micro-opticelements by empty space onto a respective plurality of photosensorelements; emitting light represented by a display signal by usingdisplay elements interleaved with the photosensor elements in anessentially planar arrangement such that the essentially planararrangement can function as a display for the display device and toperform image capture simultaneously; generating from each of thephotosensor elements a photosensor output signal in response to lightreceived thereon; receiving the photosensor output signals andperforming thereon at least one image formation operation and a digitalprocessing operation, wherein the digital processing operation includesone of an edge effect handling operation to remove edge effectsrepresented in the output signals, a sharpening operation and anaggregation operation; and outputting from the digital processingoperation respective image signals, each of which represents arespective portion of an image of the distant non-adjacent object,wherein each of the respective image signals is responsive to receivelight from the distant nonadjacent image source.
 21. The method of claim20, further comprising: arranging the plurality of photosensor elementsin a first array; scanning the photosensor output signals generated bythe photosensor elements on a line-by-line basis; and performing thedigital image processing operation on photosensor output signalsgenerated during the line-by-line photosensor scan.